Papers with uncertainty estimation

46 papers
On the Calibration and Uncertainty of Neural Learning to Rank Models for Conversational Search (2021.eacl-main)

Copied to clipboard

Challenge: Existing methods to rank documents in decreasing order of their probability of relevance are not well calibrated and have several sources of uncertainty.
Approach: They propose to calibrate deterministic neural rankers for conversational search problems . they then use two techniques to model the uncertainty of neural ranker's uncertainty .
Outcome: The proposed rankers output a predictive distribution of relevance as opposed to point estimates.
Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questions (2024.eacl-long)

Copied to clipboard

Challenge: Using model uncertainty as supervision for deciding when to ask may not be the most effective way to resolve model uncertainty.
Approach: They propose to generate clarification questions based on model uncertainty estimation and compare it to several alternatives to generate questions .
Outcome: The proposed approach improves the model uncertainty of a collaborative dialogue task and shows that it is more effective than other alternatives.
Efficient Out-of-Scope Detection in Dialogue Systems via Uncertainty-Driven LLM Routing (2025.acl-industry)

Copied to clipboard

Challenge: Out-of-scope (OOS) intent detection is critical in task-oriented dialogue systems . without effective OOS detection, such inputs could lead to incorrect responses, reduced user trust, and eventual system failures.
Approach: They propose a modular framework that combines uncertainty modeling with fine-tuned large language models (LLMs) their method yields state-of-the-art results on key OOS detection benchmarks .
Outcome: The proposed framework yields state-of-the-art results on key OOS detection benchmarks including real-world OOS data.
Multi-Task Dense Retrieval via Model Uncertainty Fusion for Open-Domain Question Answering (2021.findings-emnlp)

Copied to clipboard

Challenge: Existing approaches to multitask dense retrieval are not effective due to corpus inconsistency.
Approach: They propose to train individual dense passage retrievers for different open-domain question-answering tasks and aggregate their predictions during test time.
Outcome: The proposed method achieves state-of-the-art performance on 5 benchmark QA datasets, with up to 10% improvement in top-100 accuracy compared to a joint-training multi-task DPR on SQuAD.
Uncertainty Estimation and Reduction of Pre-trained Models for Text Regression (2022.tacl-1)

Copied to clipboard

Challenge: State-of-the-art classification and regression models are often not well calibrated and can be inaccurate.
Approach: They quantify calibration of pre- trained language models for text regression . they apply uncertainty estimates to augment training data in low-resource domains .
Outcome: The proposed model calibrations improve performance and generalizability in low-resource settings.
Extracting or Guessing? Improving Faithfulness of Event Temporal Relation Extraction (2023.eacl-main)

Copied to clipboard

Challenge: Existing models for event temporal relation extraction are based on data-driven machine learning . however, TEMPREL extraction is not accurate under distribution shifts.
Approach: They propose to conduct counterfactual analysis to attenuate the effects of two types of training biases: the event trigger bias and the frequent label bias.
Outcome: The proposed model extracts TempRel and timelines more faithfully compared to SOTA methods . it is based on two perspectives: one is to extract genuinely based upon contextual description . the other is to provide proper uncertainty estimation and abstain from extraction when no relation is described in the text .
Test-time Adaptation for Machine Translation Evaluation by Uncertainty Minimization (2023.acl-long)

Copied to clipboard

Challenge: evaluators of machine translation systems often use text-based metrics to evaluate performance . however, these metrics lack semantic-level information and exhibit poor correlation with human ratings . authors propose a method to reduce inference bias of neural metrics in out-of-distribution data .
Approach: They propose to reduce inference bias by using uncertainty estimation, test-time adaptation, and inference to reduce model uncertainty.
Outcome: The proposed method reduces model uncertainty and improves correlation performance across models.
Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models (2025.emnlp-main)

Copied to clipboard

Challenge: Existing approaches to quantify uncertainty are limited in vision-language models . however, current models display notable miscalibration across diverse tasks and settings .
Approach: They evaluate verbalized confidence in vision-language models using visual reasoning . they propose a prompting strategy that improves confidence alignment in multimodal settings .
Outcome: The proposed method improves confidence alignment across multimodal settings.
Calibration vs Decision Making: Revisiting the Reliability Paradox in Unlearned Language Models (2026.acl-srw)

Copied to clipboard

Challenge: Model calibration is commonly used as a proxy for reliability, but low calibration error does not necessarily imply reliable decision rules.
Approach: They investigate the calibration error gap in generative language models using the TOFU benchmark and attribution-based shortcut detection.
Outcome: The proposed model calibrations achieve low calibration error compared to pretrained models and retain low calibration despite reduced accuracy on the forget split.
E-NER: Evidential Deep Learning for Trustworthy Named Entity Recognition (2023.findings-acl)

Copied to clipboard

Challenge: Named entity recognition (NER) systems focus on improving model performance, ignoring the need to quantify model uncertainty.
Approach: They propose to introduce two uncertainty-guided loss terms to the conventional EDL and a series of uncertainty-guiding training strategies to solve these challenges.
Outcome: The proposed method achieves better OOV/OOD detection performance and generalization ability on OOV entities compared to state-of-the-art methods.
Handling Ambiguity in Emotion: From Out-of-Domain Detection to Distribution Estimation (2024.acl-long)

Copied to clipboard

Challenge: Experimental results show that incorporating utterances without majority-agreed labels into an additional class reduces the classification performance of the other emotion classes.
Approach: They propose to combine utterances without majority-agreed labels into an additional class . they propose to quantify uncertainty in emotion classification using evidential deep learning .
Outcome: The proposed method retains classification accuracy while effectively detects ambiguous emotion expressions.
ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation (2023.emnlp-main)

Copied to clipboard

Challenge: Recent studies indicate that learning from individual annotations outperforms learning from aggregated labels, though they require a considerable amount of annotation.
Approach: They propose to use a multi-head model to learn from disagreements in an active learning setting to identify annotators with a single head.
Outcome: The proposed model outperforms a single-head model in terms of uncertainty estimation and prediction while saving 70% of the annotation budget.
How Certain is Your Transformer? (2021.eacl-main)

Copied to clipboard

Challenge: Obtaining reliable uncertainty estimations for such neural networks (NNs) is challenging due to the huge number of parameters in these deep learning models.
Approach: They propose to use Monte Carlo dropout to estimate uncertainty for Transformer-based models and construct inexpensive estimates using Determinantal Point Processes.
Outcome: The proposed estimates improve the quality of detection of error-prone instances.
Uncertainty Estimation on Sequential Labeling via Uncertainty Transmission (2024.findings-naacl)

Copied to clipboard

Challenge: Named entity recognition tasks are often suboptimal for NER . previous work focused on UE-NER, which estimates uncertainty scores for ner .
Approach: They propose to use a Sequential Labeling Posterior Network to estimate uncertainty for NER . they propose to consider wrong-span cases and to evaluate the specificity of wrong-pan cases.
Outcome: The proposed system improves on three datasets and AUPR on MIT-Restaurant datasets.
Extracting Chemical-Protein Interactions via Calibrated Deep Neural Network and Self-training (2020.findings-emnlp)

Copied to clipboard

Challenge: Several natural language processing methods have been used to extract interactions between chemicals and proteins from biomedical text data.
Approach: They propose a method to extract chemical–protein interactions from biomedical text data . they use a pre-trained language-understanding model and calibration techniques to estimate uncertainty .
Outcome: The proposed approach achieves state-of-the-art performance on the Biocreative VI ChemProt task while preserving higher calibration abilities.
STAR: Constraint LoRA with Dynamic Active Learning for Data-Efficient Fine-Tuning of Large Language Models (2024.findings-acl)

Copied to clipboard

Challenge: Existing studies show that supervised training is still necessary for complex reasoning tasks.
Approach: They propose a method to integrate uncertainty-based active learning and LoRA to effectively integrate the two methods.
Outcome: The proposed approach outperforms baseline models on three reasoning tasks.
Benchmarking Uncertainty Metrics for LLM Target-Aware Search (2025.findings-emnlp)

Copied to clipboard

Challenge: Existing uncertainty metrics for LLM search methods do not capture the diverse types of uncertainty needed to guide different optimization goals.
Approach: They propose a framework for uncertainty benchmarking that captures four different uncertainty types . the uncertainty types Answer, Correctness, Aleatoric, and Epistemic serve different optimization goals .
Outcome: The proposed framework identifies four different uncertainty types . the uncertainty types serve different optimization goals in LLM search .
Ambiguity Meets Uncertainty: Investigating Uncertainty Estimation for Word Sense Disambiguation (2023.findings-acl)

Copied to clipboard

Challenge: Existing supervised methods treat word sense disambiguation as a classification task but ignore uncertainty estimation (UE) in the real-world setting, the data is always noisy and out of distribution.
Approach: They propose to use word sense disambiguation to determine an appropriate sense for a word given its context to determine the most appropriate sense.
Outcome: The proposed model reflects data uncertainty satisfactorily but underestimates model uncertainty.
Efficient Nearest Neighbor based Uncertainty Estimation for Natural Language Processing Tasks (2025.findings-naacl)

Copied to clipboard

Challenge: Existing methods for uncertainty estimation are inadequate for safety-critical applications.
Approach: They propose a method that uses the distances from neighbors and the ratio of labels in neighbors to estimate uncertainty.
Outcome: The proposed method outperforms baseline and density-based methods in calibration and uncertainty metrics.
Mirroring Users: Towards Building Preference-aligned User Simulator with User Feedback in Recommendation (2026.acl-long)

Copied to clipboard

Challenge: Large Language Models lack specific task alignment and large-scale simulations are challenging due to their ambiguity, noise and massive volume.
Approach: They propose a framework that leverages user feedback in RSs with advanced LLM capabilities to generate high-quality simulation data.
Outcome: The proposed framework boosts the alignment with human preferences and in-domain reasoning capabilities of the fine-tuned LLMs.
NeuroSym-Cal: Bridging the Reasoning-Execution Gap in Code Generation via Hierarchical Calibration (2026.findings-acl)

Copied to clipboard

Challenge: Existing calibration methods rely on the assumption that consensus implies correctness . Existing methods fail under systematic errors, leading to miscalibrated high-confidence predictions.
Approach: They propose a hierarchical calibration framework that measures confidence at two levels . they propose sensitivity analysis to measure local curvature of deductive process .
Outcome: The proposed framework de-saturates overconfident errors and improves selective generation performance on OOD benchmarks.
Should We Trust This Summary? Bayesian Abstractive Summarization to The Rescue (2022.findings-acl)

Copied to clipboard

Challenge: Xu et al., 2019; Lewis e t al, 2019) show that Bayesian summarization methods can generate high quality summaries but suffer from a couple of issues when inputs lie far from the training data distribution.
Approach: They propose to extend state-of-the-art summarization models with Monte Carlo dropout and perform multiple stochastic forward passes to approximate Bayesian inference.
Outcome: The proposed method outperforms deterministic summarization models on multiple benchmark datasets.
Semi-Supervised Dialogue Abstractive Summarization via High-Quality Pseudolabel Selection (2024.naacl-long)

Copied to clipboard

Challenge: Semi-supervised dialogue summarization (SSDS) leverages model-generated summaries to reduce reliance on human-labeled data.
Approach: They propose a scoring approach that encapsulates three primary dimensions of summarization model quality.
Outcome: The proposed method reduces reliance on human-labeled data and improves the performance of summarization models.
Active Learning for Abstractive Text Summarization (2022.findings-emnlp)

Copied to clipboard

Challenge: Abstractive text summarization (ATS) requires a long document and short summaries.
Approach: They propose a query strategy for AL in abstractive text summarization that uses uncertainty estimation to reduce model performance.
Outcome: The proposed query strategy improves ROUGE and consistency scores for annotated datasets . it also increases the performance of the model, compared to passive annotation.
Accounting for Sycophancy in Language Model Uncertainty Estimation (2025.findings-naacl)

Copied to clipboard

Challenge: Effective human-machine collaboration requires machine learning models to externalize uncertainty.
Approach: They propose a generalization of the definition of sycophancy bias and a new algorithm to account for scophancies in uncertainty estimation.
Outcome: The proposed algorithm can account for sycophancy in uncertainty estimation process.
ADDMU: Detection of Far-Boundary Adversarial Examples with Data and Model Uncertainty Estimation (2022.emnlp-main)

Copied to clipboard

Challenge: Existing methods show poor performance under Far Boundary (FB) adversarial examples.
Approach: They propose to use a new technique to detect adversarial examples based on data and model uncertainty to outperform existing methods.
Outcome: The proposed method outperforms existing methods by 3.6 and 6.0 AUC points under each scenario.
Uncertainty-aware Parameter-Efficient Self-training for Semi-supervised Language Understanding (2023.findings-emnlp)

Copied to clipboard

Challenge: Existing methods for pre-trained language models rely on noisy data, which can be expensive if all parameters are updated.
Approach: They propose a self-training framework that incorporates Monte Carlo dropouts into the model and judiciously selects reliable pseudo-labeled examples based on confidence and certainty.
Outcome: The proposed framework improves performance and efficiency over multiple tasks over multiple datasets.
Uncertainty Quantification of Large Language Models through Multiple Uncertainty Sources (2026.findings-acl)

Copied to clipboard

Challenge: Existing methods for uncertainty quantification fail to capture multifaceted nature of natural language generation.
Approach: They propose a multi-resource Uncertainty Quantification framework that integrates heterogeneous uncertainty signals into a unified measure.
Outcome: The proposed framework outperforms existing methods on CoQA, NQ_Open, and HotpotQA.
Boosting Cross-Lingual Transfer via Self-Learning with Uncertainty Estimation (2021.emnlp-main)

Copied to clipboard

Challenge: Recent pre-trained language models have achieved remarkable zero-shot performance . we propose a self-learning framework that utilizes unlabeled data of target languages .
Approach: They propose a self-learning framework that utilizes unlabeled data of target languages to select silver labels for cross-lingual transfer tasks.
Outcome: The proposed framework outperforms baseline models on two cross-lingual tasks by 10 F1 on average and 2.5 accuracy on natural language inference (NLI).
Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments (2022.findings-emnlp)

Copied to clipboard

Challenge: Neural scaling laws define a predictable relationship between a model’s parameter count and its performance after training in the form of a power law.
Approach: They perform an empirical investigation of language understanding tasks and evaluate their results to determine whether scaling laws can be used to accelerate model development.
Outcome: The proposed scaling laws can be exploited for debugging convergence when training large models, and can predict the performance of larger models.
Hybrid Uncertainty Quantification for Selective Text Classification in Ambiguous Tasks (2023.acl-long)

Copied to clipboard

Challenge: Existing methods for text classification tasks are inherently ambiguous and can cause errors.
Approach: They propose a method that combines epistemic and aleatoric uncertainty to estimate toxicity detection errors.
Outcome: The proposed method outperforms existing methods for toxicity detection and other ambiguous text classification tasks.
Harnessing the Power of Large Language Model for Uncertainty Aware Graph Processing (2024.lrec-main)

Copied to clipboard

Challenge: Existing methods for graph processing rely on assumptions about data relations that are inadequate when handling large and complex graph data.
Approach: They propose a large language model enhanced by an uncertainty-aware module to provide a confidence score on the generated graph data.
Outcome: The proposed approach surpasses state-of-the-art algorithms by a substantial margin on ten datasets.
Decoding Uncertainty: The Impact of Decoding Strategies for Uncertainty Estimation in Large Language Models (2025.findings-emnlp)

Copied to clipboard

Challenge: Decoding strategies affect the probability distribution underlying the output of a language model and can therefore affect both generation quality and uncertainty.
Approach: They investigate the impact of decoding strategies on uncertainty estimation in large language models .
Outcome: The proposed methods improve the uncertainty estimation of large language models by reducing repetition.
Can We Trust the Performance Evaluation of Uncertainty Estimation Methods in Text Summarization? (2024.emnlp-main)

Copied to clipboard

Challenge: Text summarization is a key natural language generation task, but the high cost of inaccurate summaries raises concerns about the reliability of uncertainty estimation on text summarisation (UE-TS) evaluation methods.
Approach: They propose a UE-TS benchmark that evaluates the uncertainty estimation capabilities of two large language models and one pre-trained language model on three datasets.
Outcome: The proposed benchmark evaluates the uncertainty estimation capabilities of two large language models and one pre-trained language model on three datasets, with human-annotation analysis incorporated where applicable.
Trick or Neat: Adversarial Ambiguity and Language Model Evaluation (2025.findings-acl)

Copied to clipboard

Challenge: Direct prompting fails to detect ambiguity while linear probes can decode ambiguities with high accuracy, sometimes exceeding 90%.
Approach: They introduce an adversarial ambiguity dataset that includes syntactic, lexical, and phonological ambiguities along with adversarials.
Outcome: The proposed dataset includes syntactic, lexical, and phonological ambiguities along with adversarial variations.
Adaptive Platt Scaling with Causal Interpretations for Self-Reflective Language Model Uncertainty Estimates (2025.findings-emnlp)

Copied to clipboard

Challenge: a new study examines the ability of large language models to self-monitor and ask for human intervention.
Approach: They propose a formal analysis of LLM self-reflection for uncertainty estimation using domain adaptation theory.
Outcome: The proposed method improves accuracy and human interpretation on reasoning tasks.
Are Data Augmentation Methods in Named Entity Recognition Applicable for Uncertainty Estimation? (2024.emnlp-main)

Copied to clipboard

Challenge: Named Entity Recognition (NER) is a key task in NLP to find mentions of named entities and classify them into predefined categories.
Approach: They investigated the impact of data augmentation on confidence calibration and uncertainty estimation in Named Entity Recognition (NER) tasks.
Outcome: The data augmentation improves calibration and uncertainty in cross-genre and cross-lingual setting, especially in-domain setting.
A Survey of Uncertainty Estimation Methods on Large Language Models (2025.findings-acl)

Copied to clipboard

Challenge: Large language models (LLMs) have demonstrated remarkable capabilities but could produce biased, hallucinated, or non-factual responses.
Approach: They propose to conduct extensive experimental evaluations of LLM uncertainty estimation methods . large language models have demonstrated remarkable capabilities across tasks .
Outcome: The proposed method could produce biased, hallucinated, or non-factual responses . a lack of comprehensive surveys on LLM uncertainty estimation is a problem .
The Role of Model Confidence on Bias Effects in Measured Uncertainties for Vision-Language Models (2025.findings-emnlp)

Copied to clipboard

Challenge: Quantifying epistemic uncertainty in open-ended tasks is challenging due to the presence of aleatoric uncertainty, which arises from multiple valid answers.
Approach: They conduct experiments on visual question answering tasks and find that mitigating prompt-introduced bias improves uncertainty quantification.
Outcome: The proposed approach reduces uncertainty quantification in visual question answering tasks by mitigating prompt-introduced biases.
Towards Harmonized Uncertainty Estimation for Large Language Models (2025.acl-long)

Copied to clipboard

Challenge: Large language models (LLMs) have demonstrated exceptional capabilities in handling a wide range of downstream tasks.
Approach: They propose a method that employs a lightweight model trained on data aligned with the target LLM’s performance to adjust uncertainty scores.
Outcome: The proposed method achieves improvements of up to 60% over existing methods.
CoverICL: Selective Annotation for In-Context Learning via Active Graph Coverage (2024.emnlp-main)

Copied to clipboard

Challenge: In-context learning (ICL) uses few-shot labeled examples to perform selective annotation.
Approach: They propose an algorithm that incorporates uncertainty sampling into selective annotation for ICL . CoverICL builds a nearest-neighbor graph based on the semantic similarity between candidate ICL examples .
Outcome: The proposed algorithm outperforms existing methods for low-budget active learning (AL) it is up to 2x more budget-efficient than SOTA methods for high-budge AL.
DebUnc: Improving Large Language Model Agent Communication With Uncertainty Metrics (2025.findings-emnlp)

Copied to clipboard

Challenge: Multi-agent debates can improve the accuracy of Large Language Models by having multiple agents discuss solutions over several rounds of debate.
Approach: a debate framework that uses uncertainty metrics to assess agent confidence is proposed . the framework uses textual prompts or a modified attention mechanism that adjusts token weights .
Outcome: The proposed framework assesses agent confidence using uncertainty metrics . the framework is available at https://github.com/lukeyoffe/debunc.
Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systems (2025.findings-emnlp)

Copied to clipboard

Challenge: Existing approaches to selecting reliable responses from multiple LLMs often depend on external verifiers, human evaluators, or self-consistency techniques.
Approach: They propose a calibrated log-likelihood-based selection framework to improve multi-LLM performance.
Outcome: The proposed method outperforms majority voting and exceeds self-consistency performance when using a large number of model calls.
DUD: Decoupled Update Dynamics for Reliable Uncertainty Quantification in Large Language Models (2026.acl-long)

Copied to clipboard

Challenge: Accurate Uncertainty Quantification (UQ) is critical for reliable deployment of Large Language Models (LLMs).
Approach: They propose a framework that explicitly decouples FFN and Attention contributions via noise-induced causal interventions to capture model's internal fragility.
Outcome: The proposed framework outperforms state-of-the-art baselines in both uncertainty estimation and calibration while exhibiting superior cross-dataset generalization.
SUE: Sparsity-based Uncertainty Estimation via Sparse Dictionary Learning (2025.emnlp-main)

Copied to clipboard

Challenge: Existing methods to estimate uncertainty use predictive confidence, structural characteristics of representation space, or stochastic variation in model outputs.
Approach: They propose a new uncertainty estimation framework based on sparse dictionary learning by identifying dictionary atoms associated with misclassified samples.
Outcome: The proposed framework outperforms or matches existing methods on several NLU benchmarks and sentiment analysis benchmarks.
Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs (2026.findings-acl)

Copied to clipboard

Challenge: Large Language Models exhibit systematic biases across demographic groups.
Approach: They propose to use auditing as uncertainty estimation over a fairness metric . they propose to introduce the Bounded Active Fairness Auditor for query-efficient auditing .
Outcome: The proposed auditing tool reduces query access costs and improves performance over time.

What is GenGO?

GenGO is an NLP powered publication search system. It currenctly indexes 30k+ papers from ACL Anthology, and implements multi-aspect summarization, semantic search, and more!

Information

About
Limitations